1. Field of the Invention
This invention generally relates to the field of data processing and more particularly to high-performance sorting of data in computer systems.
2. Description of the Related Art
Prior art methods for sorting data sets which may be too large to fit within the memory of a computer generally involve a process of sort-merging, wherein data is sorted in units as large as available memory can handle (the xe2x80x9cinternalxe2x80x9d sort), output as xe2x80x9cstringsxe2x80x9d, and then the strings are xe2x80x9cmerged,xe2x80x9d by successive operations if necessary, until the result is a single fully sorted string.
There is a vast body of technology which has developed in connection with sorting and merging. Much of this technology concerns improved methods for internally sorting, improved methods for merging, and improved methods of communicating with disk drives and other I/O devices during the sort merge process. Early work in this area in the context of disk-based sort-merge operations is reflected in the commonly assigned patent, U.S. Pat. No. 4,210,961.
Making the usual assumption that one will not be writing output over the input file, it follows that at a minimum, the sort-merge process involves copying the input to the output. Where the input resides in a disk file, part of this process is simply reading the entire input file from disk. However, relatively little of the prior research activity in this area has been directed to the manner of reading the input file.
Conventionally, the input file in a sort-merge process is accessed in accordance in the normal manner provided by the operating system, in which data is read from the disk in the logical order of file contents. The actual physical blocks of data on the disk corresponding to each file are not, however, generally stored in a contiguous or linear order. In practice, there is considerable physical discontinuity of recorded data blocks, both within individual files, and from file to file in a disk file system. Indeed, even if linearly recorded at the outset, the data blocks of files in a production computer system may become highly fragmented as blocks are read, revised and written over the course of normal usage. Even a newly created file may be fragmented if its data is larger than the next free spaces made available by the operating system. In normal operation, the operating system takes care of this, maintaining a directory which keeps track of the correspondence between the blocks of data that comprise a file, and the physical location of each block on the storage media. Yet in most operating systems the physical order of blocks is generally allowed to become discontinuous and fragmented.
The result of this disorder and fragmentation of raw disk data is that the process of reading files using normal operating system calls (or any other disk access methods that operate similarly) generally results in significant disk read head repositioning during the read operation. Since this mechanical movement can be the slowest operation on the computer, sometimes by orders of magnitude, reading a disk in this manner can be highly inefficient, and the delays involved can be significant, even compared to the time required to completely sort random file contents. A sort-merge job that is constrained to read the disk in this manner will thus necessarily suffer from this significant inefficiency. Considerable improvement in sort-merge operations can be obtained if this inefficiency can be overcome.
Accordingly, it is an object of the present invention to provide an improved sort-merge method which avoids the inefficiencies of normal disk access during input. Among the objects of the present invention, therefore, are the following:
To reduce disk read time by reducing the amount of head repositionings necessary to read the sort input;
To achieve such reduction by performing sequential rather than random reads of the input file, to the extent feasible; and
Despite having read the file in a physical sequential order, being able to keep track of the logical sequence of blocks as well, so that aspects of the original record order can be maintained in the sorted output if so specified by the user.
The foregoing and other objects of the invention are accomplished by taking into consideration the fact that records can be read in any order if they are subsequently to be sorted. Thus, input from disk can be processed by reading the working directory maintained by the operating system to determine all of the blocks associated with the input data to be sorted. The data block identities so determined are sorted in accordance with their physical location on the disk, thereby providing a sequential order for reading. The input data is read in this sequential order, and then, using largely conventional methods, sorted into one or more strings and merged as necessary to form the fully sorted output. Since the original record order in the file is known from the working directory that has been read, that order can be utilized if and as necessary, for example to preserve the original order of records with equal keys.
The manner in which the invention achieves these objects is more particularly shown by the drawings enumerated below, and by the detailed description that follows.